Hierarchical Confidence Based Clustering

In order to ease the complexity of asking one learner to accomplish the task of prediction, multiple learners are often used. Our approach towards this is to split the data set into clusters based on the confidence a learner can classify them. This repeats until some predetermined hierarchical level. Each cluster of data then gets its own specialized learner. This will allow the algorithm to concentrate more on the data points that it is unsure of, resulting in a higher confidence rate on them. We then use a centroid and neural network to learn which data points go into which clusters such that new data points can be matched up with the best classifier. We have applied this approach to the problem of financial classification for the Dow Jones Industrial Average, as well as handwritten digit recognition. Our hypothesis is that this approach can be used to create a committee of learners, each specialized towards a subset of the data space. Our results show that we can improve upon traditional strategies for each data set, while offering the ability to parallelize learning.